Abstract:

Semantic representation is how our brains process and understand the meaning of words and information from other sources, facilitating functions such as speech production and memory retrieval. In the evolution of how we understand semantic representation, two distinct viewpoints emerge. The traditional cognitive perspective, exemplified by linguistic symbol representation, posits that linguistic symbols serve as vessels for semantic representation without sensory attributes. According to this view, conceptual information travels through propositional networks established among linguistic symbols and is processed through a logic akin to computer operations to achieve semantic representation (Markman & Dietrich, 2000). Conversely, the embodied cognitive viewpoint, typified by perceptual symbol representation, contends that semantic representation relies on perceptual symbols imbued with embodied (or multimodal) traits. This perspective suggests that rich sensory and emotional experiences actively participate in the representation process through perceptual simulation, endowing these representations with characteristics akin to perceptual images (Barsalou, 1999). The perspective of symbolic integration provides insights for understanding the semantic representation characteristics in foreign language learning.

Currently, there is a lack of a thorough comparison regarding how the two different systems for characterizing information impact language learning. The precise influence of these various representation methods on learning foreign language vocabulary and their corresponding cognitive neural mechanisms remains unclear. In an effort to clarify the influence of semantic representation on vocabulary acquisition, this study employs a paradigm focused on vocabulary memory recognition. During the memory encoding phase, participants are exposed to linguistic (native language) and nonverbal (directional space and motion cues) schematic materials, prompting them to engage with the language symbols and perceptual symbols associated with foreign language vocabulary. This research compares how these two symbol representations affect the processes of both encoding and recognizing foreign language vocabulary using both behavioral observations and EEG technique, shedding light on the cognitive processes and neural mechanisms linked to semantic representation in vocabulary acquisition.

Initially, during the memory encoding stage, the study examines how the two symbol representations impact the depth of vocabulary encoding by concentrating on the late positive components (LPC) triggered by the encoding process. Subsequently, during the recognition stage, the research investigates how the two ways of symbol representations impact the extraction of vocabulary meaning at a visual level, specifically focusing on the N400 components evoked during vocabulary recognition. Furthermore, to gauge the level of perceptual simulation exhibited by the two symbol representation modes in the recognition stage, the study scrutinizes time-frequency analysis of EEG activity during the recognition process, analyzing the rhythmic oscillations of brain μ waves in the 8-13 Hz frequency band. Lastly, to assess how the two representation patterns influence memory retrieval performance, the study examines the impact of vocabulary recognition on memory retrieval by investigating power variations in the frequency band (4-7 Hz) within the theta range.

A total of 52 participants were randomly assigned to the two symbolic representation conditions and executed a vocabulary learning-recognition task. To manipulate the participants’ semantic representations effectively when learning foreign language vocabulary, two types of spatial semantics, i.e., “up” and “down,” were chosen as the learning materials. Furthermore, to investigate the learning process effectively, the foreign words expressing the meanings of “up” and “down” were selected from languages that the participants were completely unfamiliar with. In the learning stage, spatial cues (Figure 1A) or Chinese characters (Figure 1B) of “up” and “down” were presented first as semantic priming stimuli to initiate the participants’ linguistic symbolic or perceptual symbolic representations of the foreign words subsequently presented. The participants were then required to learn the foreign words. After every 20 words learned, a test on the semantic recognition of the foreign words was conducted. Behavioral and EEG data were collected to investigate the different effects of linguistic symbolic and perceptual symbolic representations on the learning and recognition stages.

Behavioral results. Independent sample t-tests were used to analyze the judgment of learning (JOL) scores and recognition accuracy of participants under the two conditions. The findings revealed that participants' recognition accuracy (Figure 2B) was significantly higher under perceptual symbol representation conditions (0.74 ± 0.10, mean ± standard deviation) compared to linguistic symbol representation conditions (0.67 ± 0.11), t (50) = 2.53, p = 0.015, 95% CI [0.02,0.13], d = 0.68. However, there was no significant difference in JOL scores (Figure 2A) between perceptual (5.87 ± 1.05) and linguistic (5.83 ± 1.20) symbol representation conditions, t (50) = 0.14, p = 0.89.

EEG time-domain analysis. In the memory encoding stage, we performed a repeated measurement analysis of variance (ANOVA) for representation type (perceptual/linguistic symbol) × electrode position on the average amplitude of LPC (Figure 3). The results indicated that the interaction between these factors was not significant, F(4, 200) = 0.07, p = 0.87. However, the main effect of the representation type was significant, F(1, 50) = 4.85, p = 0.032, 95% CI [0.13,2.80], η²_p = 0.09. This suggests that the average amplitude of LPC under perceptual symbol representation conditions (5.64 ± 2.23 μV) was significantly higher than that in the linguistic symbol representation condition (4.17 ± 2.56 μV). Additionally, there was a significant main effect of the electrode, F(4, 200) = 129.78, p < 0.001, η²_p = 0.72. In the memory retrieval stage, we conducted a repeated measurement ANOVA for representation type and electrode position on the average amplitude of N400 induced by correctly recognizing old items (Figure 4). The results revealed that the interaction between these two factors was not significant, F(4, 200) = 0.79, p = 0.40. However, the main effect of the representation type was significant, F(1, 50) = 5.08, p = 0.029, 95% CI [0.17,2.98], η²_p = 0.09. This indicates that the average amplitude of N400 under perceptual symbol representation conditions (5.26 ± 2.13 μV) was significantly more pronounced than that in the linguistic symbol representation condition (6.84 ± 2.86 μV). The main effect of the electrode was not significant, F(4, 200) = 1.31, p = 0.26.

EEG time-frequency domain analysis. Firstly, the assessment of event-related spectral perturbation (ERSP) results within the μ range for correctly identified previous items demonstrate that compared to the linguistic symbolic representation condition, greater inhibition of EEG rhythmic oscillations in the μ frequency band was observed in the perceptual symbolic representation condition, resulting in lower power (Figure 5, dashed line boxes). Specifically, the repeated measurement ANOVA for representation type and electrode position indicated that the main effect of the representation type [F(1, 50) = 25.60, p < 0.001, η²_p = 0.34] and the electrode [F(4, 200) = 9.31, p = 0.001, η²_p = 0.16] were significant. Additionally, the interaction between the two factors was also significant, F(4, 200) = 4.76, p = 0.018, η²_p = 0.09. Further simple effect analysis indicates that at each electrode position, the power within the μ band in the perceptual symbol representation are significantly lower than those in the linguistic symbol representation condition, ps ≤ 0.001. Secondly, the assessment of ERSP results within the theta range demonstrated perceptual symbol representation results in a more pronounced frontal θ power enhancement (Figure 5, solid line boxes). Specifically, the repeated measurement ANOVA for representation type and electrode position indicated that the main effect of the representation type is marginally significant, F(1, 50) = 3.60, p = 0.064, while the main effect of electrode is significant, F(4, 200) = 20.16, p < 0.001, η²_p = 0.29. Additionally, the interaction between the two factors was also significant, F(4, 200) = 6.61, p = 0.004, η²_p = 0.12. Further simple effect analysis indicates that the frequency power within theta band in the frontal (a), mid-frontal (b), and central (c) region are significantly higher under perceptual symbol representation conditions than those in the linguistic symbol representation condition, F_a(1, 50) = 8.17, p = 0.006, 95% CI [0.06,0.33], η²_p = 0.14; F_b(1, 50) = 6.66, p = 0.013, 95% CI [0.04,0.30], η²_p = 0.12, F_c(1, 50) = 4.29, p = 0.043, 95% CI [0.004,0.26], η²_p = 0.08. No significant differences are found in other locations, ps > 0.05.

To sum up, the event related potential results showed that during the learning stage, the perceptual symbolic representation induced more positive LPC components in the time window of 400~800 ms) than the linguistic symbolic representation condition. During the recognition stage, in relation to the linguistic symbolic representation condition, the perceptual symbolic representation evoked larger N400 components in the time window of 200~400 ms after the onset of the recognition words. The results of EEG time- frequency analysis showed that during the recognition stage, the perceptual symbolic representation condition elicited lower μ band power and higher θ band power than the linguistic symbolic representation condition (the time windows of the two bands were 200~800 ms after the onset of the recognition words).

In conclusion, results indicated that compared with linguistic symbolic representation, perceptual symbolic representation had a delayed influence on vocabulary encoding. It promoted deep encoding processing of vocabulary and improved the efficiency of vocabulary semantic retrieval through perceptual simulation in the recognition process, thereby implicitly improving the semantic recognition of vocabulary.